Online Performance Management Using Hybrid Reinforcement Learning

We present a new hybrid approach to performance management, combining disparate strengths of Reinforcment Learning (RL) with model-based (e.g. queuing-theoretic) approaches. Our method trains nonlinear function approximators using offline RL on data collected while a model-based policy controls the system. By training offline we avoid potentially poor performance in live online training, while function approximation allows generalization across both states and actions, so that the need for exploratory actions may be greatly reduced. Our results show that, in a prototype resource allocation scenario among multiple web applications, hybrid RL training can achieve significant performance improvements over a variety of initial queuing model-based policies. We also find that, as expected, RL can deal effectively with both transients and switching delays, which lie outside the scope of traditional steady-state queuing theory.